TACTICAL  ACTION  OFFICER 
INTELLIGENT  TUTORING  SYSTEM  (TAO  ITS) 


Richard  H.  Stottler 
Stottler  Henke  Associates,  Inc. 
1660  So.  Amphlett  Blvd.,  Suite  350 
San  Mateo,  CA  94402 
(650) 655-7249 


LCDR  Michael  Vinkavich 
Surface  Warfare  Officers  School  Command 
446  Cushing  Road 
Newport,  RI 02841 
(401)  841-4963 

Abstract 

The  U.S.  Navy's  Surface  Warfare  Officers  School  (SWOS)  in  Rhode  Island  is  pioneering  the  use  of  a  low- 
cost  simulation-based  intelligent  tutoring  system  (ITS)  as  part  of  its  Tactical  Action  Officer  (TAO)  training 
program  to  train  Navy  officers  in  high-level  tactical  skills.  This  software  was  designed  and  built  for  SWOS 
for  use  on  standard  PCs,  was  introduced  to  the  School  in  early  1999,  and  the  Navy  has  a  royalty-free  license 
to  use  it.  The  software  can  be  used  both  as  a  classroom  aid  and  by  individual  students.  A  key  objective  of 
the  software  is  to  increase  the  active  training  that  officers  receive  to  improve  their  ability  to  apply  their 
conceptual  knowledge  of  tactics.  Early  results  from  its  use  with  two  classes  are  encouraging  and  indicate 
that  the  software  will  succeed  in  this  goal  by  enabling  as  much  as  a  ten-fold  increase  in  hands-on  training. 

The  software  has  three  parts.  First,  there  is  a  scenario  generator,  with  which  instructors,  with  limited 
assistance  from  a  programmer,  can  create  any  number  of  simulated  scenarios.  These  can  be  set  in  any  part 
of  the  world,  and  populated  with  different  surface  and  air  platforms.  Each  individual  platform  is 
implemented  as  an  "intelligent  agent"  and  can  be  given  its  own  performance  characteristics  and  behaviors 
so  it  will  act  realistically.  For  example,  a  hostile  plane  will  have  its  own  mission  (which  the  student  can 
only  surmise)  but  will  react  to  challenges  from  the  student's  friendly  platforms.  Second,  there  is  the 
intelligent  tutoring  system,  which  presents  selected  scenarios  to  the  student  to  practice  different  tactical 
concepts.  The  software  will  adaptively  select  scenarios  for  individual  students,  that  practice  concepts  he  or 
she  hasn't  yet  practiced  or  has  recently  failed,  or  enable  a  student  to  pick  any  scenario.  As  well  as  the 
intrinsic  feedback  that  free-play  simulations  naturally  provide  a  student,  the  TAO  ITS  provides  detailed, 
useful  extrinsic  feedback  to  the  student  once  a  scenario  is  finished,  which  reviews  all  the  concepts 
attempted  and  whether  they  were  passed  or  failed.  At  this  point,  the  student  can  review  multimedia  material 
about  any  concept,  or  see  a  replay  of  the  scenario  to  review  errors.  The  third  part  of  the  software  is  an 
instructor  interface  tool  for  instructor  to  review  all  the  students'  work  with  the  tutoring  system  to  assess 
their  progress  in  detail. 

This  paper  describes  the  Tactical  Action  Officer  Intelligent  Tutoring  System  as  an  example  of  what  ITSs 
can  do  and  the  benefits  they  can  provide.  It  also  includes  an  explanation  of  why  the  case-based  reasoning 
technique  was  used  in  the  software  to  reduce  three  problems  commonly  associated  with  intelligent  tutoring 
systems:  effective  incorporation  of  subject  expert  knowledge  in  the  software,  cost,  and  development  time.  It 
also  reviews  SWOS's  experience  with  the  software  since  its  introduction,  students'  opinions  of  the  software, 
and  suggests  ways  in  which  future  simulation-based  intelligent  tutoring  systems  might  be  improved  based 
on  SWOS's  experience. 
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INTRODUCTION 

The  mission  of  the  Surface  Warfare  Officers^ 
School  (SWOS)  in  Newport,  Rhode  Island  is  to 
provide  professional  education  and  training  to 
prepare  officers  of  the  U.S.  Surface  Navy  to 
serve  at  sea.  The  importance  of  this  mission  was 
succinctly  emphasized  by  ADM  Arleigh  Burke 
who  said,  "This  ship  is  built  to  fight.  You  had 
better  know  how." 

As  part  of  his  training  at  SWOS,  each  Surface 
Warfare  Officer  learns  how  to  "fight"  his  ship  as 
a  Tactical  Action  Officer.  The  TAO  training 
consists  of  three  months  of  classroom  and 
simulator  time  wherein  students  are  exposed  to 
all  elements  of  surface  warfare;  air,  surface, 
subsurface,  and  amphibious  operation  as  well  as 
electronic  and  other  support  mechanisms.  One 
major  responsibility  of  a  TAO  is  to  be  able  to  be 
able  to  exercise  command  over  the  major 
systems  of  his  ship  (weapons,  support  platforms, 
radar  and  sonar,  and  navigation)  during 
potentially  hostile  situations.  The  tactical 
decisions  he  makes  during  such  situations  can 
easily  affect  the  outcome  of  the  ship's  mission,  as 
well  as  have  life  or  death  consequences. 

The  U.S.  Navy  is  continually  reviewing  and 
improving  its  training  program  to  prepare  its 
young  surface  warfare  officers  for  this 
considerable  responsibility  so  that  they  will 
perform  effectively  the  first  time  when  they  are 
put  to  the  test  in  action.  Such  training  is  very 


expensive  (one  recent  estimateMaf  the  total  cost 
to  train  a  surface  warfare  officer  is  over 
$80,000),  requires  a  high  instructor/student  ratio, 
and  the  cost  is  exacerbated  by  loss  of  SWOs 
from  the  Navy. 

The  Navy  has  been  using  desktop  training 
software  to  improve  training  through  more  active 
simulations  without  increasing  the  instructor  to 
student  ratio,  thereby  reducing  overall  cost 
without  reducing  training  effectiveness.  Such 
software  might  also  prove  invaluable  at  times  of 
emergency  when  the  U.S.  needs  to  rapidly 
expand  its  fleet.  During  a  period  of  rapid 
expansion,  this  program  would  maintain  the 
same  training  standards  throughout  a  wide  range 
of  instructional  sources;  that  is,  if  such  an 
expansion  were  to  occur,  SWOS  could  not 
accommodate  the  increased  throughput  and  other 
instructional  resources  would  have  to  offer  this 
type  of  training. 

A  promising  piece  of  software  introduced  to 
SWOS  in  early  1999  has  been  the  Tactical 
Action  Officer  Intelligent  Tutoring  System, 
designed  and  built  to  SWOS  specifications. 

TACTICS  TRAINING  FOR 
TAOs 

The  first  formalized  tactical  training  any  Surface 
Warfare  Officer  receives  begins  with  the 
Department  Head  TAO  course.  (Some  students 
are  more  proficient  in  tactics  due  to  billets  held 
as  division  officers.) 
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Currently,  TAO  students  spend  approximately 
forty  (40)  hours  in  simulators  designed  to  allow 
practical  application  of  classroom  instruction. 
The  future  of  the  navy  requires  fully  qualified 
individuals  when  they  report  to  their  ship, 
requiring  longer  periods  of  effective  realistic 
training.  TAO  ITS  is  a  step  in  the  right  direction 
to  help  fill  the  gap. 


INTELLIGENT  TUTORING 
SYSTEMS 

Purpose  and  Nature  of  Intelligent 
Tutoring  Systems 

Intelligent  tutoring  systems  (ITS)  can  best  be 
defined  as  advanced  training  software  that 
mimics  a  human  tutor  by  adapting  its 
instructional  approach  to  each  individual  student. 
They  are  particularly  valuable  for  teaching 
complex  cognitive  tasks  such  as  trouble 
shooting,  problem  solving,  and  resolving  critical 
situations.  As  a  human  tutor  does,  an  ITS 
continually  monitors  and  assesses  each  student's 
actions,  infers  the  student's  state  of  knowledge, 
and  decides  on  the  next  instructional  event  to 
maximize  the  student's  learning.  To  do  this  in  a 
significant  and  cost-effective  way,  intelligent 
tutoring  systems  use  artificial  intelligence. 

One-on-one  tutoring  by  skilled  human  tutors  is 
widely  regarded  as  the  single  best  mode  of 
instruction.  A  study4>y  Benjamin  S.  Bloom  of 
the  University  of  Chicago  and  Northwestern 
University  concluded  that,  under  the  best 
learning  conditions  they  could  devise  (tutoring 
one-on-one),  the  average  student  was  2  Sigma 
above  the  average  control  student  taught  under 
conventional  group  methods  of  instruction.  That 
is,  the  average  tutored  student  was  above  98%  of 
the  students  in  the  control  class. 

Conventional  computer-based  training  (CBT) 
software  is  not  designed  to  provide  such  a  high 
level  of  adaptive  response  to  each  individual 
student  as  an  individual  human  tutor  or  ITS  can. 
In  fact,  most  CBT  software  more  closely 
resembles  an  "electronic  textbook"  rather  than  an 
"electronic  teacher."  Just  as  a  book  implicitly 
encourages  a  student  to  start  at  the  front  and 
move  to  the  back,  such  CBT  software  usually 
encourages  a  student  to  move  linearly  through  a 
set  of  multimedia  material,  with  the  occasional 
multichoice  questions  to  test  the  student's 


retention  of  the  information.  Such  tests  do  not 
assess  the  student's  ability  to  apply  the 
information;  and  it  is  the  ability  to  apply 
information  in  a  job  that  training  should  be  really 
about. 

Also,  conventional  computer-based  training  is 
not  able  to  meaningfully  incorporate  use  of  free- 
play  simulators  into  their  curriculum,  affordably. 
This  is  a  major  shortcoming  of  conventional 
CBT  as  student  manipulation  of  sophisticated 
simulators  that  realistically  replicate  issues  that 
they  will  encounter  on  the  job  is  widely 
recognized  to  be  a  highly  effective  training 
technique.  The  catch  has  been  that  simulators 
without  instructors  are  virtually  useless  for 
training,  and  their  unsupervised  use  can  even 
result  in  students  learning  misinformation. 
Students  working  on  simulators  need  instructors 
to  point  out  their  correct  and  incorrect  actions,  to 
brief  them  in  context  on  the  underlying  concepts 
that  are  being  taught,  and  to  decide  on  the  next 
appropriate  simulated  scenario  for  the  student  to 
run. 

Intelligent  tutoring  systems  are  ideal  for 
incorporating  desktop  free-play  simulators  into 
computer-based  training  since  the  software  can 
stand  in  for  a  human  tutor  in  all  his  roles. 

Existing  CBT  course  material  can  often  be 
integrated  with  ITS-enabled  simulator  and  other 
active  training.  In  this  way,  ITS  technology  can 
greatly  leverage  the  training  value  of  existing 
CBT  and  desktop  simulators.  As  shown  in  the 
Figure  1,  the  ITS  can  monitor  a  student's 
interaction  with  both  simulation  and  other 
training  content  in  the  CBT,  create  and  update 
the  student  model  and  decide  on  the  next 
instructional  event  (e.g.,  provide  a  hint,  ask  a 
question,  run  a  new  scenario,  display  multimedia 
to  explain  a  concept,  alert  a  human  instructor  that 
the  student  needs  special  help,  etc.). 

To  keep  track  of  each  student,  the  intelligent 
tutoring  system  creates  and  maintains  a  "student 
model"  for  each  individual  from  the  first  time  he 
or  she  logs  onto  the  software.  Depending  on  the 
sophistication  of  a  particular  intelligent  tutoring 
system,  the  student  model  keeps  different 
amounts  of  information  on  the  student.  The  most 
basic  information  includes  those  tasks  the 
student  has  performed,  and  his  or  her 
performance  on  those  tasks,  but  not  just  a  record 
of  their  actions.  From  this  information,  the 
software  might  estimate  the  student's  mastery  of 
relevant  skills  and  knowledge,  and  the  student's 
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ability  to  apply  them  when  appropriate.  For 
example,  a  student  may  be  able  to  apply  a 
concept  in  one  set  of  circumstances,  but  not 
under  other  circumstances,  so  it  is  important  that 
the  software  tests  the  student's  knowledge  of 
each  concept  under  different  circumstances. 


Student 


Figure  1.  ITS  can  integrate  free-play  simulators 
and  multimedia  course  material 


Design  of  Intelligent  Tutoring 
Systems 

As  well  as  a  student  model,  intelligent  tutoring 
systems  usually  have  a  number  of  other 
functional  parts. 

There  is  an  instructor  or  pedagogical  module, 
which  contains  the  knowledge  of  "how  to  teach." 
Decisions  about  timing  of  instructional  events 
are  originated  here  in  response  to  input  from  the 
student  model 

A  domain  knowledge  or  expert  module  contains 
the  knowledge  that  is  being  taught.  Effective 
representation  of  this  knowledge,  particularly 
concepts  and  mental  models,  is  a  challenging  job 
for  the  ITS  designer.  In  the  case  of  the  Tactical 
Action  Officer  Intelligent  Tutoring  System, 
much  of  this  knowledge  is  embedded  in  the  rules 
driving  the  simulation  and  underlying  finite  state 
machines  that  recognize  when  a  student's  actions 
indicate  observance  or  non-observance  of  a 
concept  that  is  being  taught  or  tested  in  a 
particular  scenario. 


There  is  also  a  communications  module,  which 
controls  the  interactions  with  the  user,  and  is 
responsible  for  presenting  the  information  to  the 
student  in  the  most  effective  way. 

There  are  different  ways  of  representing  the 
various  forms  of  knowledge  that  are  needed  for 
an  ITS,  and  the  software  designer  is  responsible 
for  selecting  a  way  that  best  suits  the  knowledge 
to  be  taught,  the  pedagogical  methods  to  be 
implemented,  and  the  goals  of  the  ITS,  not  least 
of  which  is  to  enable  a  cost-effective  software 
product.  Designing  an  ITS  is  still  a  complex 
process  and  depends  on  the  skill  and  experience 
of  the  designers. 

One  approach  to  designing  intelligent  tutorinu 
systems  is  to  use  case-based  reasoning  (CBR)‘r 
The  first  advantage  of  this  approach  is  that 
students  learn  best  from  examples  (e.g., 
scenarios  simulating  realistic  situations),  and 
their  abilities  are  best  tested  in  the  realistic 
contexts  provided  by  these  scenarios.  A  case- 
based  approach  also  reduces  the  costly  and 
timely  activity  of  developing  an  explicit  expert 
model  that  can  perform  at  the  required  level  of 
expertise,  which  is  a  problem  in  developing  a 
rule-based  intelligent  tutoring  system.  A  related 
benefit  of  the  approach  is  that  it  reduces  the 
effort  of  eliciting  the  knowledge  from  the  subject 
expert.  It  is  usually  much  easier  for  an  expert,  an 
experienced  tactical  action  officer  for  example, 
to  convey  his  knowledge  and  reasoning  via  cases 
rather  than  rules.  He  is  usually  more  responsive 
if  asked  to  describe  an  actual  specific  problem 
and  its  solution,  with  an  explanation  of  the  steps 
required  to  produce  the  solution  rather  than 
express  the  same  knowledge  in  a  set  of 
consistent  and  complete  rules. 


History  and  Effectiveness  of 
Intelligent  Tutoring  Systems 

Although  the  history  of  intelligent  tutoring 
systems  goes  back  around  30  years,  the  first 
commercial  ITSs,  which  teach  high-school 
algebra  and  geomeiry,  were  introduced  by 
Carnegie  Learning  ctround  1998.  They  have 
been  extensively  tested  in  schools  with 
extremely  good  results.  For  example,  students 
taught  using  their  Algebra  I  product: 

•  Score  50-100%  higher  on  problem  solving 
and  multiple  representation  tests  than  a 
control  group; 
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•  Perform  15-25%  better  than  control  classes 
on  standardized  tests; 

•  Are  twice  as  likely  to  complete  Geometry 
and  enroll  in  Algebra  II;  and 

•  Are  more  likely  to  attend  class. 

A  high  school  teacher  k\vh  o  helped  develop  the 
software  and  continues  to  use  it,  has  observed  a 
phenomenon  of  possibly  greater  relevance  to  the 
needs  of  the  military.  One  of  the  best 
consequences  of  using  the  software  is  that 
students  who  pass  algebra  tests  are  now  more 
likely  to  be  able  to  put  algebra  to  work  in  solving 
their  practical  problems.  Before  using  the 
software,  a  smaller  proportion  of  passing 
students  were  inclined  or  able  to  actually  use 
what  they  had  learned,  indicating  that 
conventional  classroom  teaching  resulted  in 
more  inert  knowledge. 

Although  intelligent  tutoring  systems  are  only 
now  being  commercialized,  there  have  been  a 
number  of  systems  either  prototyped  or  fully 
developed  for  the  U.S.  government—  including 
the  Tactical  Action  Officer  Intelligent  Tutoring 
System.  Unfortunately,  there  have  been  few 
performance  assessments  of  ITSs  in  service, 
though  the  few  that  have  been  done  have  been_ 
very  positive.  Dr.  Wes  Regian  and  co-authors^-1 
analyzed  the  effectiveness  of  233  conventional 
computer-based  training  courses  and  3  intelligent 
tutoring  systems.  Student  learning  when  using 
the  courseware  was  compared  with  student 
learning  under  similar  circumstances  but  without 
the  courseware.  The  conventional  software 
produced  a  significant  positive  effect,  but  the 
impact  of  the  ITSs  was  more  than  twice  as 
positive. 

TACTICAL  ACTION  OFFICER 
INTELLIGENT  TUTORING 
SYSTEM 

Description  of  the  TAO  ITS 

The  TAO  ITS  is  a  good  example  of  a  “general 
class”  of  useful  training  systems  -  simulation- 
based  intelligent  tutoring  systems  designed  to 
run  on  a  PC.  It  has  been  used  at  the  Navy's 
Surface  Warfare  Officer's  School  (SWOS)  in 
Rhode  Island  since  early  1999.  As  well  as  being 
a  powerful  assistant  to  the  classroom  instructor, 
the  TAO  ITS's  advanced  capabilities  as  an 
"electronic  teacher"  enable  a  student  to  use 


simulations  for  learning  on  his  own,  anytime, 
anyplace. 

The  TAO  ITS  was  designed  to  provide  tactical 
action  officer  students  at  SWOS  with  realistic, 
practice-based  instruction  and  individualized 
feedback.  A  tactical  action  officer  controls  his 
ship's  sensors  and  weapons  and  directs  the 
movements  of  the  ship  and  other  support  vessels 
and  aircraft.  The  TAO  also  monitors  the 
movements  and  actions  of  friendly  and  enemy 
ships,  planes,  missiles,  and  submarines  in  the 
region.  The  TAO  integrates  this  information  in 
real  time  to  form  a  dynamic  tactical  picture, 
selects  appropriate  responses,  and  issues  orders. 

The  TAO  ITS  allows  students  to  use  information 
presented  in  class  and  receive  individual 
feedback  as  to  their  performance  in  tactical 
decision-making,  use  of  ship’s  sensors  and 
weapon  systems,  and  reporting  procedures.  It 
allows  instructors  to  review  these  student  runs 
and  tailor  scenarios  to  help  overcome  any 
observed  deficiencies  in  their  understanding 
thereby  ensuring  classroom  training  is  adequate. 
ITSs  can  allow  the  student  to  provide  feedback 
to  the  instructor  in  the  scenario  itself.  For 
example,  if  the  ITS  stated  that  the  student  failed 
a  principle  that  differed  from  one  that  he  had 
been  taught,  he  would  be  able  to  imbed  this 
information  within  the  scenario  for  instructor 
review. 

When  the  TAO  ITS  is  sent  to  the  fleet,  the  ships 
will  be  able  to  create/tailor  scenarios  specific  to 
each  ship's  mission.  This  will  give  the 
commanding  officers  another  tool  to  help  fine 
tune  skills  and  procedures  required  for  the  ship’s 
missions. 

The  TAO  ITS  software  enables  students  to  act  as 
TAOs  in  simulations  of  tactical  scenarios,  as  a 
conventional  training  simulator  might  do.  Unlike 
conventional  training  simulators,  after  a  student 
completes  each  scenario,  the  TAO  ITS  also 
automatically  evaluates  the  student's  actions  to 
determine  tactical  principles  that  the  student  has 
correctly  applied  or  failed  to  apply.  These 
detailed  assessments  of  student  performance  are 
available  to  both  the  student  and  his  instructor. 

This  evaluation  is  carried  out  using  sophisticated 
pattern-matching  algorithms  defined  by  tactical 
experts  via  graphical  user  interface,  without 
programming.  The  student  can  then  learn  how  to 
correct  his  problems  by  either  selecting 
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multimedia  training  material  associated  with  any 
principle,  or  by  replaying  relevant  parts  of  the 
last  scenario  he  worked  to  review  his  mistakes. 

The  TAO  ITS  also  helps  the  student  choose  the 
next  scenario  to  practice  on.  The  student  can 
allow  the  software  to  choose  a  scenario  that 
contains  untested  principles,  other  scenarios  that 
test  principles  recently  failed  by  the  student,  or 
simply  pick  his  or  her  own  preferred  scenario. 
The  instructor  can  use  a  scenario  generator 
included  in  the  software  package  to  create  any 
number  of  additional  scenarios,  defining 
complex  behaviors  for  each  friendly  and  enemy 
ship  and  aircraft  to  create  realistic,  multi-agent 
tactical  simulations. 

The  software  has  three  parts:  the  scenario 
generator  with  which  instructors,  with  limited 
assistance  from  a  programmer,  can  create  any 
number  of  simulated  scenarios;  the  student 
interface,  which  presents  selected  scenarios  to 
the  student  to  practice  different  tactical  concepts; 
and  the  instructor  interface  tool  for  the  instructor 
to  review  all  the  students'  work  with  the  tutoring 
system  and  to  assess  their  progress  in  detail. 


Scenario  Generator 

The  scenarios  created  with  the  scenario  generator 
can  be  set  in  any  part  of  the  world,  and  populated 
with  different  surface,  air  and  underwater 
platforms  (i.e.,  ships,  planes,  helicopters, 
missiles,  submarines).  See  Figure  2  for  the  main 
interface  of  the  Scenario  Generator.  Each 
individual  platform  is  implemented  as  an 
"intelligent  agent"  and  can  be  given  its  own 
performance  characteristics  and  behaviors  (see 
Figures  3  and  4).  For  example,  a  hostile  plane 
will  have  its  own  mission  such  as  flying  various 
patterns  to  search  out  enemy  vessels,  and  when 
one  is  found  to  attack  it.  The  student  can  not  see 
these  defined  behaviors  and  therefore  can  only 
surmise  the  goals  of  hostile  platforms  and  their 
likely  reactions  to  challenges  from  the  student's 
forces. 


Figure  2.  Scenario  Generator 


Figure  3.  Tool  to  edit  a  platform,  in  this  case  a 
MIG-29 
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Editing  MIG  29. 
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Figure  4.  By  pressing  "Type"  button  in  Figure  3, 
this  second-level  tool  to  set  the  basic 
performance  characteristics  of  the  chosen 
platform  is  made  available. 


recognition  of  observance  of  one  principle,  or 
that  one  evaluated  sequence  may  indicate  that 
several  principles  were  breached. 

Student  Interface 

The  heart  of  the  intelligent  tutoring  system  is  the 
student  interface  that  presents  selected  scenarios 
to  the  student  in  order  to  practice  different 
tactical  concepts.  The  software  was  designed  to 
adaptively  select  scenarios  for  individual 
students,  who  need  to  practice  concepts 
(principles)  he  or  she  has  not  yet  practiced  or 
ones  he  has  recently  failed.  It  enables  a  student 
or  instructor  to  pick  any  scenario  from  all  of  the 
ones  available.  As  well  as  the  intrinsic  feedback 
that  free-play  simulations  naturally  provide,  the 
TAO  ITS  provides  detailed,  useful,  extrinsic 
feedback  to  the  student  once  a  scenario  is 
finished.  This  feedback  reviews  all  the  concepts 
attempted  and  whether  they  were  passed  or 
failed.  At  this  point,  the  student  can  review 
multimedia  material  about  any  concept  or  see  a 
replay  of  the  scenario  to  review  errors. 


Key  to  the  training  effectiveness  of  a  scenario  is 
the  instructor's  ability  to  select  concepts  (also 
know  as  principles)  from  a  set  of  available 
concepts  in  order  to  associate  with  a  particular 
scenario.  (This  association  is  done  using  the 
Instructor  Interface  Tool,  see  below).  The 
selected  concepts  are  those  in  which  the 
instructor  intends  the  student  to  test  his  or  her 
mastery  in  the  specific  scenario.  Since  the 
simulator  is  free-play,  there  is  no  guarantee  that 
any  particular  concept  will  actually  be  tested 
when  a  student  runs  a  scenario.  For  example,  if 
the  student  orders  his  ship  to  head  away  from  an 
enemy  plane  and  remains  concealed  from  it,  an 
entirely  different  set  of  events  may  play  out  than 
those  that  would  if  the  ship  were  discovered  by 
the  plane.  To  deal  with  this  aspect  of  free-play 
simulators,  the  TAO  ITS  has  "evaluators" 
associated  with  the  concepts.  An  evaluator  is 
designed  to  look  for  prescribed  sequences  of 
events  and  actions  during  a  scenario.  For 
example,  if  a  missile  is  fired  at  the  student's  ship, 
there  may  be  a  range  of  appropriate  actions  he  or 
she  could  take  in  response.  A  number  of 
evaluators  could  be  set  up  to  examine  the  chosen 
actions,  and  depending  on  what  they  observe,  the 
software  may  recognize  that  different  principles 
are  observed  or  not.  There  is  not  a  one-to-one 
correspondence  between  evaluators  and 
principles,  and  a  combination  of  evaluated 
sequences  may  need  to  happen  to  trigger 


The  student  (or  instructor)  starts  a  session  by 
logging  on.  Once  logged  on,  the  student  can 
choose  to  review  his  or  her  previous 
performance,  review  multimedia  instructional 
material,  or  run  scenarios.  This  is  initiated  by 
picking  an  appropriate  scenario  from  those 
available  using  the  window  shown  in  Figure  5. 


Figure  5.  Screen  providing  assistance  to  student 
to  decide  which  scenario  to  run  next. 


Pressing  "OK"  in  the  "Select  Scenario  Options" 
window  launches  the  scenario  as  shown  in 
Figure  6.  The  software  is  set  to  run  a  scenario  for 
up  to  60  minutes  however  the  student  can 
terminate  the  scenario  at  any  time. 

While  there  is  a  basic  physical  simulation 
driving  all  the  platforms  in  a  scenario,  the 


Page  8 


EIT026 


simulator  in  the  TAO  ITS  is  inherently 
conceptual.  For  example,  a  tactical  action  officer 
on  an  AEGIS  cruiser  works  in  the  Command 
Information  Center  (CIC),  and  is  supported  by  a 
large  number  of  individuals  who  provide  him 
with  information  and  respond  to  the  TAO's 
orders.  To  simulate  all  of  the  commands,  the 
TAO  ITS  provides  the  means  to  implement 
them.  Thus,  the  left  uppermost  section  of  the 
screen  operates  all  the  ship's  weapon  systems, 
the  lower  left  section  issues  commands  to  any 
supporting  aircraft,  the  upper  right  section 
provides  control  over  the  ship's  navigation,  and 
the  lower  right  section  operates  the  radar  and 
sonar  equipment  and  displays  numerical 
responses  to  this  equipment.  The  lower  middle 
panel  displays  communications  from  crew 
members,  for  example,  reports  of  incoming 
missiles,  and  these  communications  can  also  be 
heard  with  the  voice  synthesizer.  The  central 
display  panel  is  a  reasonable  physical  simulation 
of  display  in  the  CIC,  and  it  displays  only  the 
information  that  would  be  realistically  available 
to  the  TAO  at  any  time. 

An  open  question  for  advanced  training  systems 
is  to  what  extent  physical  realism  in  the 
simulator  is  important  to  meeting  the  training 
objectives.  The  higher  the  degree  of  physical 
realism  that  is  used,  the  greater  the  expense  of 
creating  the  hardware  and  software  and  in 
updating  it  as  the  platform  changes.  The  TAO 
ITS  was  designed  primarily  to  teach  cognitive 
skills.  The  reaction  of  students  and  instructors  to 
it  indicate  strongly  that  lack  of  physical  realism 
did  not  negatively  impact  meeting  its  training 
objectives. 


Figure  6.  Main  Student  Interface. 

Once  terminated,  the  software  evaluates  the 
student’s  actions  by  comparing  his  actions  and 
the  circumstances  under  which  he  made  those 


actions  against  the  "evaluations,"  and  prepares  an 
Evaluation  Summary  (see  Figure  7).  Doing  the 
evaluation  for  the  whole  scenario  session  was  a 
design  choice.  For  other  ITS  applications,  it  may 
be  more  appropriate  to  do  an  on-going  evaluation 
that  could  generate  hints,  provide  hints,  or  alter 
the  course  of  the  session  in  some  other  chosen 
way. 


Figure  7.  Evaluation  Summary 

The  Evaluation  Summary  lists  the  situations  in 
which  the  student  demonstrates  understanding  of 
concepts  (principles)  by  their  correct  application. 
Correct  decisions  are  marked  by  a  green  circle, 
and  situations  where  understanding  of  concepts 
was  not  observed  (either  by  incorrect  or  omitted 
actions)  are  marked  by  red  circles.  Also 
provided  are  the  time  and  description  of  the 
action.  It  also  provides  additional  information 
that  an  instructor  might  mention  to  a  student  in 
regular  one-on-one  tutoring.  The  exact 
principles  observed  or  not  observed  for  any  of 
these  situations  can  be  found  by  clicking  on  the 
particular  situation.  By  clicking  on  any  noted 
principle  the  student  will  be  taken  to  multimedia 
information  that  explains  the  principle.  The 
Evaluation  Summary  form  also  allows  the 
student  to  replay  the  recent  scenario  from  the 
start  or  from  a  selected  time. 


Instructor  Interface  Tool 

The  instructor  can  manage  the  activities  of  all 
students  using  this  tool  (see  Figure  8).  It 
provides  the  instructor  with  tools  to  manage  the 
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hierarchy  of  instructional  principles  and  the  set 
of  multimedia  review  content;  to  link  specific 
multimedia  review  content  to  principles  so  that 
when  a  student's  actions  endanger  friends  the 
review  text  covers  that  topic;  and  to  associate 
principles  and  evaluations  with  specific 
scenarios. 


Figure  8.  Instructor  Interface  Tool 


Ongoing  improvements  being 
made  to  the  TAO  ITS 

Work  continues  on  additional  improvements  to 
the  current  TAO  ITS  to  enable  trainers  to 
implement  a  wider  range  of  teaching  strategies. 
An  ITS  authoring  tool  is  being  built  that  works 
with  the  TAO  ITS  or  other  similar  training 
simulators.  With  the  authoring  tool,  a  trainer  will 
be  able  to  specify  a  hierarchy  of  principles  to  be 
taught,  identifying  prerequisite  and  dependent 
principles,  so  that  the  software  can  determine 
their  order  of  presentation.  Alternatively  the 
instructor  can  specify  the  desired  order  in  which 
to  deliver  instruction.  He  will  also  be  able  to 
instruct  the  software  to  implement  a  wide  range 
of  teaching  strategies  to  be  used  with  beginner 
through  advanced  students,  as  appropriate.  In 
addition,  a  trainer  will  be  able  to  instruct  the 
software  how  he  wishes  it  to  perform  initial 
expertise  and  ongoing  expertise  assessments  of 
students  at  different  levels  of  performance. 

Based  on  these  assessments,  the  software  will 
continually  regrade  students  so  they  always  are 
taught  the  appropriate  material  in  an 
instructionally  effective  way,  based  on  past 
performance  and  the  system’s  estimates  of  their 
knowledge,  ability  and  learning  style.  The 
authoring  tool  will  also  give  the  trainer  more 
control  over  the  way  scenarios  will  be  selected. 


For  example,  a  beginner  may  have  scenarios 
selected  for  him  according  to  a  trainer-specified 
criterion,  while  an  advanced  student  may  be  able 
to  continue  selecting  the  scenarios  he  prefers  to 
run. 

With  these  improvements,  an  ITS  can  deliver  an 
even  greater  one-on-one  learning  experience  for 
each  student,  enabling  the  power  of  simulation- 
based  training  to  be  provided,  without  requiring 
the  presence  of  a  human  instructor. 


EARLY  REACTION  TO  TAO 
ITS  BY  SWOS  INSTRUCTORS 
AND  STUDENTS 

Overview  of  Dr.  Joan  Johnston's 
survey 

A  survey  of  student  reaction  to  the  TAO  ITS  was 
conducted  at  SWOS  in  lune  1999.  The  survey 
was  designed  by  Dr.  Joan  H.  Johnston  of  the 
Naval  Air  Warfare  Center  Training  Systems 
Division  (NAWCTSD)  in  Orlando,  Florida.  Dr. 
Johnston  also  evaluated  its  results. 

SWOS  department  head  students  in  one  of 
LCDR  Black’s  classes  participated  in  a 
demonstration  of  the  TAO-ITS.  Following  the 
classroom  demonstration,  LCDR  Black 
distributed  the  survey.  Twelve  students 
responded. 

(LCDR  Black  was  integral  to  the  development  of 
TAO-ITS  and  its  introduction  into  the 
curriculum  at  SWOS.  Additionally,  TAO-ITS 
was  used  more  often  than  just  for  the  survey.  It 
has  been  an  integral  portion  of  the  AEGIS 
Tactical  Employment  Training  portion  of  the 
TAO  curriculum  for  several  classes.) 

The  survey  was  designed  to  gauge  student 
reaction  to  the  TAO  ITS  in  two  prospective 
roles — first,  as  a  classroom  aid;  and  second,  in 
its  more  demanding  role  as  an  intelligent  tutor  to 
support  the  individual  in  learning  and  practicing 
tactical  decision-making  tasks. 

Results  of  survey 

Nine  of  the  12  students  had  an  extremely 
favorable  overall  reaction  to  the  TAO  ITS  as  a 
classroom  aid,  two  had  a  favorable  reaction,  and 
only  one  student  was  neutral.  All  of  the 
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responding  students  had  an  overall  favorable 
reaction  to  the  TAO  ITS  as  a  stand-alone  training 
tool,  and  80%  of  these  students'  reactions  were 
extremely  favorable.  LCDR  Gene  Black,  Lead 
AEGIS  Instructor  at  the  Surface  Warfare 
Officers  School  (SWOS)  shared  their 
enthusiasm:  "TAO  ITS  gives  a  student  tactical 
action  officer  ten  times  the  tactical  decision¬ 
making  opportunity  [compared  with  that 
provided  by]  existing  training  systems." 


CONCLUSIONS 

The  initial  reaction  from  students  to  this 
simulation-based  intelligent  tutoring  system  has 
been  very  good  and  is  indicative  of  a  strong 
positive  effect  it  has  on  learning.  By  making 
desktop  simulator-based  ITSs  more  available  to 
students  in  and  out  of  the  classroom,  cognitive 
skills  training  can  be  greatly  improved  at  a 
reasonable  (or  even  reduced)  cost.  The  intelligent 
tutoring  system  capability  removes  the  greatest 
current  limitation  on  simulator  training,  that  they 
are  virtually  useless  without  an  instructor  present 
for  evaluation,  debriefing,  remediation,  and 
scenario  selection.  An  ITS  can  be  useful  to 
instructors,  as  the  assessments  it  makes  about  the 
skills  gained  by  students  can  help  an  instructor 
plan  appropriate  classroom  activities,  and 
interventions  with  individual  students.  This  ITS 
study  also  provides  strong  support  to  the 
viewpoint  that  a  high  degree  of  physical  realism 
in  a  simulator,  which  is  expensive  to  attain,  is 
not  necessary  to  teach  cognitive  skills. 
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